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A model for desiqnlrq a writinq program evaluation 
hased on the assumptions that the entire writinq course or prc(^ram is 
evaluated, net lust student writing; that the evaluation is summative 
rather than formative: and that the role of the evaluatcr is clearly 
defined. The procedure for i nplemerting the evaluation should include 
the following steps: (1) pilot-testing the evaluation instruBent; (2) 
evaluatino student writing using hclistic scorinq or descriptive 
measures: (3) evaluating student attitudes using standard writinq 
apprehension tests, questionnaires, interviews, or 

participant-observer studies: (U) evaluatinq teacher attitudes using 
questionnaires "or Interviews supplemented by student evaluations and 
administrative observations; (5) evaluatinq 'Hdininistrator attitudes 
using • inter vie ws : (6) analyzing the transferability of "the course 
(the ability cf the course to be tauqht successfully by many 
different teachersi : (7) considering demographic information such as 
pass and retention rates: and (B)f iirplementinq a lon^^it udinal study 
cf the effects of Instruction, (AFA) 
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IN» (VIMAIION i;l Nil M (CMU:) 

In beginning to plan an evaluation, you must answer three 
Important questions. • ' 

What do i^gu want t^ evaluate? One comnion answer is 
student * writings. Various kinds of measures for evaluating 

« 

writings are available^ and I'll discuss some' of them later. But 
another possible answer (andy a better one) is this entire course 
(or program) In which students produced those writing. An 
evaluation of a course or a program would certainly include 

* 

evaluation of writings, but woufd go far beyond it. 

It's important to remember' that a program that produces 

f 

successful writing can't necessarily be called successful itself, 
for example, a course that greatly improves student writing but 
at the same time causes very high drop-out rates could hardly be 
called successful. Or what about a course that is highly 
teacher-detfendent ? A course that can be taught by onl;^ one 
teacher, or by ^nly one certain kind of teacher, doesn't seem 
^ successful as a model course. I would encourage you, then, to 



V 



consider goin^ beyond simple evaluation of student writings^ much ' 
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lore can be -lArned if you evaluate an entire course. The rest 
if my discussion ^'i^^ focus chiefly on this kind"of program 



/evaluation. 



2. HUi aofl iQL KHfiiD axfi yqai ftYaiufltina if you are 

interested in finding out more about how an exist i*ng program is 
working, in order .to improve it, then you are engaging in what is 
calle<3 \t2£lfi§ii^fi evaluation. You yourself are the audience for 
such an evaluation* 

I 

On the other hand, yoju may be interested in an evaluation for 
others (liKe administrators, or a funding agency, or a local or 

♦ 

regional education agency) . . This Kind of evaluatipn (called 

♦ * 

s^mmative evaluation) usually is used to provide infbrmation on 
which a decision about whether or not to continue a program 
(often a new or innovative one) can be based. For such an 
evaluation, it will not be enough just to describe the program; 
it should be qpmpared to other existing programs. It will very 
liKely be necessary to set up a comparison-group design for this 
kind of evaluation, ' using experimental and control groups, and 
pre- and p(^st-test measures. Such an evaluation is complex and 
expensive, but can provide a great deal' of information about how 
successful a program is. In the rest of my discussion, I'll 
assume that you are a program deyeioper who is planning, such a 
summdtive evaluation of a writing ^course. ' ' 
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!• Hilfi k£ fi:iSiy§iiDa Ai? yourself (as a' program 

'developer) are likely to have the deepest understanding of. the 
issues . involved, but it might be difficult for you tc\ be 
completely objective about the results (and it will certainly be* 
difficult for others tq believe that you can be objective) . 
Having outsiders do the evaluation will assure more objectivity^ 
but ^ will also raise the possibility of an inappropriate 
evaluation (as I will discuss shortly)/ ^ 

Once you have answered these questions, you are ready to begin, 
planning for the evaluation itself. Some words about the 
philosophy of such eval\jat^on are in order h^rfe. Evaluation of 
f educational prbgraros is never the clear-cut scientific ev^lu^tf^n 
portrayed ip statiistic^ ttxts. The variables are simply too»roany 
and too complex and too uncontrollable. The performance 'of 
students in a course is probably affected as much by politics, or 
sports schedules, qr the weather, or events in the^t (or their 
teachers •) families, 3s it is by instruction. It is certainly 
possible that the variableis most affecting the students' 
performance can*t even be identified. This means that it will be 
very difficult to reach any definite - conclusions about why 
differen.t programs produce different results. Because scientific 

* 

ver ltl cation of theories is so unlikely, the emphasis should be 
on the (jlqcQY^yy of facts. If you cah get a full and complete 
description of what actually happei)ed, it will be possible at 
least to speculate about why it happened. At the same time. 
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. there will be an Implicit suggestion that the reconstruction of 
similar-enough conditions could produce similar results; but 

.educational systems are so complex that you shouj-d not expect to 

/ 

' attain scientific proof (which implies repeatability) . 

If it is decided^that a summative evaluation of a program is 
to be carried out by an outsider, it's important for y6u as a 
' program developer tc protect your pr'ogi'am, both against an 
inappropriate evaluation (that is, measuring the .wrong' things , or 
measuring the right things v/rongly) , and against inappropriate 
conclusions from- the evalifation. The management model shown in 

■M 

Figure 1 can provide this kind of protection. 
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In this inodel, the program* deVelopers and the evaluators 
collaborate. pn planning the evaluation. The evaluators collect 
the data, and analyze it (using subcontractors if necessary). 
The prO/gram personnel ^collaborate in interpretation of the 
re»ults.\ The evaluators then write their evaluation report, and 
the pro-am developers write a s^feparate program report, 
comprehending^- the evaluation report.- This' model makes it 
'possible for program personnel (who Know most about what a. 
program is supposed to dp) to provide input at the stages Where ^ 
. tt is necessary^ but at the same time preserves the objectivity 
of the data connection and analysis. Separate reports provide 
both program * developers and evaluators with an q^pportunity to 
present their own interpretations if the results, 

o 

ERIC . \ 
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Figure 1: A Management Model for Program Evaluation 



•pr 
* per 



ograro 
sonnel 

i 
I 



evaliJation 
persoi^nel 
r 



subcontractors 



L 




I 



PLANNING 



I 
I 



COLLECTION OF DATA 



-I 



r 

ANALYSIS OF DATA 



I 



I 



I , - I 

•I INTERPRETATION OF RESULTS | 

I I 

, ^ ^ J 

I I 

I I 

I r ^« r ^ 

II I 

I EVALUATION REPORT | 



I 1 
I ^ 
I , 
I 



I 



I 



I I 
I PROGRAM REPORT | 

I _ ___J 

r 

, A U. D I E N C E 



*t enphaslze too sjtirongly that the first step in planning 
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an)' evaluation like this absolutely ifiysi be the specification of 
these^ details of the collaboration in a formal written contract. 
The preliminary discussion essential for establishfng such an 
agreement with the evaluators will clarify for both of you just 
what needs to be done, and, will also avoid potential problems by 
assigning responsibility for carrying out. the activities ^f the 
evaluation. , 

I turn now to a discussion of what measuresy'^ight be 

s' • • ■ 

appropriate to evaluation of an entire writing program* It is 

'♦important that a variety of measures be employed, foi this will 
improve the validity of the evaluation greatly (by inc^^easing the 
chances of picking up any changes that might have oci^urred) . 

To the extent possi))ltf, all evaluation measures planned should 
be pilot -tested , . either by you or by the evaluators. It is 
astonishing how many unanticipated problems can arise in an 
evaluation; pilot-testing makes it possible , to forestall at least 

# ... 

some of them.. It's useful to think of pilot-testing as a kind of 

f ■ 

formative evaluation not of the instruction but of its 

evaluation. This suggests that a .trial run of the proposed 

■ ■ '> * 
\«valuation during a previous semester can serve two purposes: as 

' a pilot-test of the proposed evaluation, and as formative 

evaluation of the program. This is . 'the most efficient way to 

pilot-test. - ' 
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1. Evaluation of siysifiDl writings is, of •curse, the most 
important single measure* Descriptive measures (such as an error 
count, a word count, or a t-unit count) might be used, ' if 
appropriate. Holistic or primary-trait, scoring can provide a 
measure of the overall quality of student writings'. A forced^ 
choice scoring offers an easy way to compare writings, but is too 
crude a measure to be very valuable by itself. A combination of 
different measures will probably be able to. provide more useful 
results than any single n\easure could. A good source for 
discussion of the various kinds of measures available is Charles 
Cooper and Lee Odell, Evaluating Wxiling (NCTEri977)* 

2. Evaluation of student attitudes is especially irapcjrtant 
when a new instructional method is being evaluated. It is 
important to investigate both st;udents' attitudes toward their 
instruction, and their attitudes toward themselves as writers. 
You should try to avoid students * becoming aware that they are in 
a special or experimental program, for this is likely to affect 
their performance as well as their attitudes. 

* I 

Pre- and post-administrations of a standardized writing 
apprehension test might be useful. 

Questionnaires are the simplest way to get . information, |'ut 
need to be used carefully.^ The language should be checked for 
clarity and appropriateness. It mi^t be difficult' to obtain 
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sufficient numbers -of responses for the results to be very 
informative. Different questions on the same subject offer a way 
to check the internal reliability of the responses, but there is 
K'always the problem of to what extent one can trust students' 
expressions of their feelings. Correlation of .students^ attitude 
responses with more objective medisures (like' their writing 
evaluation scores) might provide an index of how reliable their 
responses are. 



Interviews can provide invaluable open-ended information, but 
require skillful and objective interviewers (students must not be 
^ threatened by interviews . with anyone involved in their 
insttuction) . It may be difficult to get suitable intervicwfces , 
with accura^te. and representative perceptions abc/ut the program. 
Extreme care needs to be used in attempting to generalize about 
the information gained* from ' interviews. \ 



Important informatiop can be gained by observing students at 
work, but it is likely to be even more valuat^e to use a 
participant-observation study. *In this, a trained evaluator both 
observes students at work and participates in their work with 
" theoi. participant-observer can provide sophisticated 

^ information about both the work and the students* experiences in 
do^.ng that work; v . ' — 
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^* Te^cner att j^ tudes should also be measured > particularly if 

a reorganization of pedagogy is involved. Questionnaires and 
♦ 

interviews can be supplemented l5y student -evaluations and 
administrative observations pr evaluations (if these. exist) . 

vi 

f 

U* The attitudes of adminstrators should also be considered. 
It is important to be aware of what institutional demands (in 
terras of costs, scheduling, facilities, etc.) a program makes. 
This is information that can probably be gained only by means of 
interviews. 

5. An analysis of the transf erabil ity of the couBse is also 
often important. Can it b.e taught successfully by many different 
teachers? Is a large amount of training necessary? Comparing 
attitudes of subpopulations of teachers is one gpod way to gain 
this Rind of information; it may also be possible to break down 
other evaluation results among subpopulations (comparing the 
writing scores of one group of students to those of another, for 
example) . 

6. Dgmpgraphlc i^nformation should not be overlooked* The 
traditional demographic categories (age, gender, previous college 

history, standardized test scores', etc.) may provide a clue to 

. i, _ - - . . . 

differences in ^ results. Pass rates and retention rates are not 
usually thought of as demographic information, but ^hey aVe among 

« t ' 

the roost important pieces of information to have. They should be 

\ ^ . ■ 
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used, however, only in the context of rat^s established over 
several semesters; otherwise, there is the danger of drawing 
seriously misleading conclusions. For example, low pass rates 
may result not from an infer^r ' program, but from a certain 
teacher's. well-established gradii'ig tendencies; or spring 

.semesters may always' h^ve lower pass rates, than fall semesters; 
etc. 
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7. Finally, a ISMiiyilill^i study of the effects of the 
instruction over " time should be carried ojjt, if possible. 
Although such a study takes lots of time and mone^, it is the 
only way to determine whether a writing course has any effect 
which is more than just temporary. 
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I hope that some of thes-e suggestions will, be' able jto help you 
to make sure that evaluations of your own courses and programs 
.give you Information that is both accurate and helpful. 
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